Device and method for motion estimation and compensation

ABSTRACT

A device for motion estimation in video image data is provided. The device comprises a motion estimation unit ( 11, 21 ) for estimating a current motion vector for an area of a current image by determining a set of temporal and/or spatial candidate motion vectors and selecting a best motion vector from the set of candidate motion vectors. The motion estimation unit ( 11, 21 ) is further adapted for substantially doubling one or more of the candidate motion vectors and for including the one or more substantially doubled candidate motion vectors in the set of candidate motion vectors.

PRIORITY INFORMATION

This patent application claims priority from PCT patent applicationPCT/IB2009/054633 filed Oct. 21, 2009, which claims priority to Europeanpatent application 08167308.9 filed Oct. 22, 2008, both of which arehereby incorporated by reference.

FIELD OF THE INVENTION

The present invention relates to a device and a method for motionestimation and compensation in video image data. The invention furtherrelates to a computer program product.

BACKGROUND OF THE INVENTION

Many video processing techniques apply temporal processing, for example,for temporal predictive coding, temporal noise reduction,de-interlacing, or frame rate conversion. In all these cases, it isadvantageous that the temporal information for an area of a currentimage, for example, a block of pixels in a previous image, used in theprocessing of the current image, results from the same object, ratherthan from the same location in the previous image. Hence, it isadvantageous to compensate for the relative frame-to-frame motion of theobjects in the scene. This so-called motion compensation requires thatmotion information in video image data can be estimated using a processcalled motion estimation.

Many motion estimation algorithms used in commercial applications,especially in the consumer domain, are based on a so-called recursivesearch strategy, where a current motion vector for an area of a currentimage, for example, for a block of pixels, is determined from only alimited number of previously estimated motion vectors and, optionally,additional (pseudo random) update vectors. This is usually done by: a)calculating a set of temporal and/or spatial candidate motion vectorsfrom the limited number of previously estimated motion vectors and,optionally, the additional (pseudo random) update vectors, whereinspatial candidate motion vectors are based on previously estimatedmotion vectors in the current image and wherein temporal motion vectorsare based on previously estimated motion vectors in a previous image; b)calculating a match error, for example, a block matching error, forrespective candidate motion vectors, and; c) selecting the currentmotion vector from the set of temporal and/or spatial candidate motionvectors by comparing the match errors of the respective candidate motionvectors. One well-known example of such a recursive algorithm is the 3-Drecursive search block matching described by G. de Haan et al. in“True-Motion Estimation with 3-D Recursive Search Block Matching”, IEEETrans. on Circuits and Systems for Video Technology, Vol. 3, No. 5,October 1993, pages 368-379.

Due to their recursive structure, one of the main considerations in thedesign and application of these motion estimation algorithms is theconvergence and accuracy of the calculated motion vector field. This isusually handled by a trade-off between the composition of the set ofcandidate vectors (for example, the number of candidate vectors for ablock of pixels) and the statistical distribution of the update vectors.One can typically increase the speed of the convergence by applying morecandidate vectors, though this comes at the expense of a highercomputational load. On the other hand, a higher accuracy of the motionvector field can be achieved by using small update vectors, but thisalso slows down the convergence speed. With many of today's temporallyrecursive motion estimation algorithms, a good convergence can beobtained after processing a few images, typically in the range of 3 to5.

In order to provide a good and fast convergence of the motion vectorfield, motion estimation algorithms that use a temporal recursivestructure basically require that the temporal distance betweenconsecutive input images is largely equidistant so that the relativeframe-to-frame motion of objects in the scene (caused either by amovement of the objects or by a movement of the camera) is as far aspossible constant. However, in some video processing applications, thisrequirement is not necessarily fulfilled. For example, a wireless videotransmission to a mobile device might use a frame rate of 15 Hz, thatis, the number of input images transmitted to and processed by themobile device each second is 15. When the video is derived from a moviesource that was originally recorded with 24 images per second, thenecessary adaptation of the frame rate might be accomplished by simplyskipping individual movie images at the distribution side. The missingimages can result in irregular motion patterns in the input image datareceived by the mobile device. When applying a temporally recursivemotion estimation in the mobile device, this can result in a badconvergence of the motion estimation algorithm, resulting in unreliableand inaccurate motion vectors. The application of these motion vectorsby further video processing techniques may then result in visibleartifacts in the processed images.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a motion estimationand compensation device that is adapted for handling missing inputimages in video image data. It is a further object of the presentinvention to provide a corresponding motion estimation and compensationmethod and a computer program.

A device for motion estimation comprises a motion estimation unitestimates a current motion vector for an area of a current image bydetermining a set of temporal and/or spatial candidate motion vectorsand selects a best motion vector from the set of candidate motionvectors. The motion estimation unit is further adapted for substantiallydoubling one or more of the candidate motion vectors and for includingthe one or more substantially doubled candidate motion vectors in theset of candidate motion vectors.

In this description and in the subsequent claims, the term“substantially doubling” may particularly denote the multiplication ofeach of the individual components of a motion vector by a factor in therange between 1.7 and 2.3, more particularly by a factor in the rangebetween 1.9 and 2.1, more particularly by a factor of 2.0.

The invention relates to the observation that when individual inputimages are missing in video image data, the length of the motion vectorsbetween successive images is suddenly doubled, resulting in badconvergence when applying temporally recursive motion estimation. Byincluding one or more substantially doubled candidate motion vectors inthe set of candidate motion vectors, a good convergence of the motionestimation can be achieved even in the case of missing input images.This preserves the quality of the motion vector field, and, thus, thequality of the output images of the video processing techniques thatapply the motion vectors. Furthermore, this solution might be obtainedat minimal additional load and with basically no additional image accessbandwidth.

According to an aspect of the invention, the device further comprises amotion vector storage unit adapted for storing the current motionvector. In case that one of the one or more substantially doubledcandidate motion vectors is selected as the current motion vector, themotion estimation unit nonetheless stores the associated regularcandidate motion vector as the current motion vector in the motionvector storage unit. This provides a simple way to incorporatesubstantially doubled candidate motion vectors in the motion estimation.

According to an aspect of the invention, the device further comprises atemporal distance detection unit adapted for detecting a doubling of thetemporal distance between the current image and the previous image. Thisinformation can be utilized when using the stored current motion vectorin motion compensation.

According to an aspect of the invention, the temporal distance detectionunit is adapted for detecting a doubling of the temporal distancebetween the current image and the previous image by analyzing the numberof substantially doubled candidate motion vectors that are selected asthe current motion vector for respective areas of the current image. Bydoing so, a doubling of the temporal distance between the current imageand the previous image can be detected in a simple and efficient way.

According to a further aspect of the invention, the temporal distancedetection unit is further adapted for keeping track of the temporaldistance between successive images and for deriving a prediction of thetemporal distance between the current image and the previous image. Byusing this information, the motion estimation can be optimally tuned tothe characteristics (in terms of missing images) of the input imagedata.

According to an aspect of the invention, the motion estimation unit isfurther adapted for including the one or more substantially doubledcandidate motion vectors in the set of candidate motion vectors independence of the predicted temporal distance between the current imageand the previous image. By doing so, the computational effort requiredfor the motion estimation can be reduced without sacrificing theconvergence of the motion vector field.

According to an aspect of the invention, the motion estimation unit isfurther adapted for substantially halving one or more candidate vectorsand for including the one or more substantially halved candidate motionvectors in the set of candidate motion vectors.

In this description and in the subsequent claims, the term“substantially halving” may particularly denote the multiplication ofeach of the individual components of a motion vector by a factor in therange between 0.35 and 0.65, more particularly by a factor in the rangebetween 0.45 and 0.55, more particularly by a factor of 0.5.

By including one or more substantially halved candidate motion vectorsin the set of candidate motion vectors, the motion estimation can easilyswitch from regular candidate motion vectors to substantially doubledcandidate motion vectors (“doubling”) and from substantially doubledcandidate motion vectors back to regular candidate motion vectors(“halving”).

According to an aspect of the invention, the device further comprises amotion vector storage unit adapted for storing the current motionvector. In case that one of the one or more substantially doubled orhalved candidate motion vectors is selected as the current motionvector, the motion estimation unit stores the substantially doubled orhalved candidate motion vector as the current motion vector in themotion vector storage unit. By doing so, a substantially doubled orhalved candidate motion vector selected as the current motion vector caneasily propagate over the current image as a spatial candidate motionvector. Due to this property, potentially only one or very fewsubstantially doubled or halved candidate motion vectors need to becalculated. Furthermore, a substantially doubled or halved candidatemotion vector selected as the current motion vector can automatically beapplied in motion compensation without requiring any additionalprocessing steps.

According to a further aspect of the invention, the device comprises atemporal distance detection unit adapted for detecting a doubling orhalving of the temporal distance between the current image and theprevious image.

According to an aspect of the invention, the temporal distance detectionunit is adapted for detecting a doubling or halving of the temporaldistance between the current image and the previous image by comparingthe lengths of current motion vectors for respective areas of thecurrent image with the lengths of previous motion vectors for relatedareas of the previous image. Herein, the term “related area” may relateto the same location in the previous image or, alternatively, may relateto a location in the previous image that is shifted by the currentmotion vector with respect to a respective area of the current image. Bydoing so, a doubling or halving of the temporal distance between thecurrent image and the previous image can be detected in a simple andefficient way.

According to a further aspect of the invention, the temporal distancedetection unit is further adapted for keeping track of the temporaldistance between successive images and for deriving a prediction of thetemporal distance between the current image and the previous image. Byusing this information, the motion estimation can be optimally tuned tothe characteristics (in terms of missing images) of the input imagedata.

According to an aspect of the invention, the motion estimation unit isfurther adapted for including the one or more substantially doubled orhalved candidate motion vectors in the set of candidate motion vectorsin dependence of the predicted temporal distance between the currentimage and the previous image. By doing so, the computational effortrequired for the motion estimation can be reduced without sacrificingthe convergence of the motion vector field.

The invention also relates to a method for motion estimation in videoimage data. The method comprises the step of estimating a current motionvector for an area of a current image by determining a set of temporaland/or spatial candidate motion vectors and selecting a best motionvector from the set of candidate motion vectors. The methodsubstantially doubles one or more of the candidate motion vectors andincludes the one or more substantially doubled candidate motion vectorsin the set of candidate motion vectors.

The invention also relates to a computer program for motion estimationin video image data, wherein the computer program comprises program codefor causing a device for motion estimation as defined in claim 1 tocarry out the steps of the method for motion estimation as defined inclaim 13, when the computer program is run on a computer controlling thedevice for motion estimation.

According to a further aspect of the present invention, the device formotion estimation and compensation further comprises a motioncompensation unit adapted for compensating the motion between the areaof the current image and a corresponding area of a previous image usingthe stored current motion vector. The stored current motion vector issubstantially doubled in case that the temporal distance detection unitdetected a doubling of the temporal distance between the current imageand the previous image.

According to a further aspect of the present invention, the device formotion estimation and compensation further comprises a motioncompensation unit adapted for compensating the motion between the areaof the current image and a corresponding area of a previous image usingthe stored current motion vector.

According to a further aspect of the invention, the image processingdevice is further adapted for temporally interpolating images and foradjusting the temporal position and the number of the temporallyinterpolated images in dependence of the predicted temporal distancebetween successive images.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is further elucidated by the following Figures andexamples, which are not intended to limit the scope of the invention.The person skilled in the art will understand that various embodimentsmay be combined.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiment(s) described hereinafter. Inthe following drawings:

FIG. 1 shows exemplarily a timing diagram illustrating the temporalposition of video images in different video image sequences,

FIG. 2 shows exemplarily a block diagram of a motion estimation deviceaccording to a first embodiment of the invention,

FIG. 3 shows exemplarily a block diagram of a motion estimation deviceaccording to a second embodiment of the invention,

FIG. 4 shows exemplarily a block diagram of a motion estimation andcompensation device according to a third embodiment of the invention.

DETAILED DESCRIPTION OF THE DRAWINGS

In video processing, it is in many cases assumed that the temporaldistance between consecutive input images is equidistant. However, insome video processing applications, this assumption is not necessarilyfulfilled. A typical example of such an application is a wirelesstransmission of a video to a mobile device. Due to power and bandwidthconstraints, the wireless transmission channel might support a framerate of only 15 Hz, that is, the number of images transmitted to andprocessed by the mobile device each second is 15. When the video isderived from a movie source that was originally recorded with 24 imagesper second, the necessary adaptation of the frame rate might beaccomplished by skipping individual movie images at the distributionside and by transmitting the remaining movie images with the desiredframe rate of 15 Hz. The consequences for the input image data of themobile device can be understood by noting that in this particularexample, the temporal position of an input image in terms of theoriginal 24 Hz movie source is incremented by 24 Hz/15 Hz=1.6 for eachnext image. This is also clarified by the following Table 1, wherein theupper row denotes the temporal positions of the input images of themobile device with respect to the 24 Hz frame rate of the original moviesource and wherein the lower row denotes the respective images of theoriginal movie source that are transmitted to the mobile device, thatis, the respective images of the original 24 Hz movie source thatconstitute the 15 Hz input image data received by the mobile device.

TABLE 1 Pos (t) 1.0 2.6 4.2 5.8 7.4 9.0 10.6 12.2 13.8 15.4 17.0 18.620.2 . . . Image 1 2 4 5 7 9 10 12 13 15 17 18 20 . . .

In this example, the mobile device receives an input image sequence: 1,2, 4, 5, 7, 9, 10, 12, 13, 15, 17, 18, 20, etc., that is, the images 3,6, 8, 11, 14, 16, 19, etc. of the original movie source are skipped atthe distribution side and, thus, are missing from the input image datareceived by the mobile device. This equates to a doubling of thetemporal distance between the images pairs {2, 4}, {5, 7}, {7, 9}, {10,12}, {13, 15}, {15, 17}, {18, 20}, etc. of the received input imagesequence. When applying temporally recursive motion estimation on suchan input image sequence, the algorithm will not converge satisfactorilyand the resulting motion vector field will have a bad quality both interms of reliability and accuracy. The disclosed device and method formotion estimation and compensation are adapted for handling such missinginput images in video image data resulting in a higher-quality motionvector field.

Further advantages provided by the disclosed device and method formotion estimation, when applied in the above-described wireless videotransmission scenario, shall be explained with reference to FIG. 1. Thefirst graph 1 shows the temporal positions of the images of the original24 Hz movie source, while the second graph 2 shows the temporalpositions of the respective movie images that are transmitted to themobile device with the desired frame rate of 15 Hz. It should be noted,that the relation between the image numbers as shown in the first graph1 and the image numbers as shown in the second graph 2 are generally notknown at the receiving side. By detecting the missing images, that is,by detecting the temporal distance between a current image and aprevious image, and by keeping track of the temporal distance betweensuccessive input images, a prediction of the frame rate of the originalmovie source and the phase relation between the input images received bythe mobile device and the original movie images may be derived. This isshown in the third graph 3, where the images of the received 15 Hz inputimage sequence have been placed correctly at their original 24 Hztemporal positions. The information about the temporal distance betweensuccessive input images can be used to optimally tune the motionestimation to the characteristics (in terms of missing images) of theinput image data. Moreover, the fourth graph 4 shows that theinformation about the temporal distance between successive input imagescan also be used for an improved temporal interpolation. For example,for a temporal interpolation that may be applied in order to increasethe frame rate of the received input image sequence from 15 Hz to 48 Hzthe information about missing images can be used to properly adjust thetemporal position and the number of the temporally interpolated images.In the given example, an interpolation position distance of ½, ¼, and ¾,respectively, denotes an interpolated image at a relative temporalposition of ¼, ½, and ¾, respectively, between a previous and a currentinput image. For example, a relative temporal position of ½ betweenimages 1 and 2 denotes an interpolated image at 1.5 (with respect to the24 Hz frame rate of the original movie source), while a relativetemporal position of ½ between images 5 and 7 denotes an interpolatedimage at 5.0 (with respect to the 24 Hz frame rate of the original moviesource).

While the skipping of individual images of the original movie source atthe distribution side has been explained with a ratio of 1.6 between theframe rate of the original movie source and the frame rate of the inputimage data received by the mobile device, different ratios may alsooccur. Moreover, while the problem of missing input images has beenexplained with respect to a wireless video transmission to a mobiledevice, this does not exclude the use of the invention in otherapplications and/or other devices, for example, in TV sets or insoftware video players running on a PC. In addition, missing inputimages might also be the result of processes other than theabove-described frame rate adaptation, for example, individual imagesmight be lost or corrupted during transmission via lossy distributionchannels. Especially with the increasing popularity of Internet basedvideo content, that is, video image data that is provided on theInternet for streaming or for download, and with the spread of theabove-mentioned wireless video transmissions to mobile devices, theproblem of missing input images due to transmission errors is becomingmore and more an issue.

FIG. 2 shows exemplarily a block diagram of a motion estimation device10 for dealing with missing input images according to a first embodimentof the invention. The device for motion estimation 10 comprises a motionestimation unit 11, a motion vector storage unit 12, and a temporaldistance detection unit 13. The motion estimation unit 11 is adapted forestimating a current motion vector for an area of a current image byselecting a best motion vector from a set of temporal and/or spatialcandidate motion vectors as the current motion vector.

The selecting of the current motion vector may comprise calculatingmatch errors for the respective candidate motion vectors and choosingthe best motion vector from the set of temporal and/or spatial candidatemotion vectors by comparing the match errors of the respective candidatemotion vectors. The temporal and/or spatial candidate motion vectors aretypically calculated from a limited number of previously estimatedmotion vectors and, optionally, additional (pseudo random) updatevectors. The calculation of the match error may comprise the calculationof a block matching error, for example, a cross-correlation (CC), a sumof absolute differences (SAD), a mean-squared-error (MSE), or somecomparable error measure, if the image area for which the current motionvector is estimated is a block of pixels, but it can also comprise thecalculation of other error metrics for more general groups of pixels,for example, groups of pixels representing structural elements ofobjects within the current image. Moreover, the motion estimation doesnot necessarily have to be performed in the image domain, but may alsobe performed on a transformed version of the image.

The set of temporal and/or spatial candidate motion vectors may bedetermined by a set of candidate descriptors. Such descriptors describehow to construct a candidate motion vector. For example, in prior artsystems, a set of temporal and/or spatial candidate motion vectors canbe defined using a set of tuples {origin, location, random}, whereinpossible values for “origin” are temporal (T) or spatial (S), wherein“location” denotes the relative location of a candidate motion vectorwith respect to the current image area, for example, when the image areafor which the current motion vector is estimated is a block of pixels, avalue of (1,−1) may indicate a block that is one block below and oneblock to the left of the current block, and wherein possible values for“random” are true (T) or false (F), indication the use of a(pseudo-random) update vector. A typical example of a set of candidatedescriptors used in a prior art system would be: {S, (−1,0), F}, {S,(0,1), F}, {T, (0,0), F}, {T, (2,2), F}, {S, (−1,0), T}, {S, (0,1), T}.

According to the invention, the motion estimation unit 11 is furtheradapted for substantially doubling one or more of the candidate motionvectors and for including the one or more substantially doubledcandidate motion vectors in the set of candidate motion vectors.

In this embodiment, the including of the one or more substantiallydoubled candidate motion vectors in the set of candidate motion vectorsmay be implemented by a set of candidate descriptors of the form{origin, location, random, modifier}, wherein possible values of theadditional “modifier” are regular (R) or double (D), indicating aregular candidate motion vector or a substantially doubled candidatemotion vector. An example of a set of candidate descriptors thatincludes an additional substantially doubled candidate motion vector inthe set of candidate motion vectors would be: {S, (−1,0), F, R}, {S,(0,1), F, R}, {T, (0,0), F, R}, {T, (2,2), F, R}, {S, (−1,0), T, R}, {S,(0,1), T, R}, {S, (0,1), F, D}. However, it would also be possible toinclude one or more substantially doubled candidate motion vectorsinstead of regular candidate motion vectors in order to not increase thecomputational effort required for the motion estimation.

The motion vector storage unit 12 is adapted for storing the currentmotion vector. In case that one of the one or more substantially doubledcandidate motion vectors is selected as the current motion vector, themotion estimation unit 11 nonetheless stores the associated regularcandidate motion vector as the current motion vector in the motionvector storage unit 12. The stored current motion vector is typicallyused for motion compensation as well as for calculating a temporalcandidate motion vector for an image area in the neighborhood of thecurrent image area in a subsequent image and/or for calculating aspatial candidate motion vector for a different image area in thecurrent image.

The temporal distance detection unit 13 is adapted for detecting adoubling of the temporal distance between the current image and theprevious image. In this embodiment, the temporal distance detection unit13 is adapted for detecting a doubling of the temporal distance betweenthe current image and the previous image by analyzing the number ofsubstantially doubled candidate motion vectors that are selected as thecurrent motion vector for respective areas of the current image. Inother embodiments, however, other heuristics/methods for detecting adoubling of the temporal distance between the current image and theprevious image may be used.

In this embodiment, the temporal distance detection unit is furtheradapted for keeping track of the temporal distance between successiveimages and for deriving a prediction of the temporal distance betweenthe current image and the previous image. In other embodiments, however,the temporal distance detection unit 13 does not have to be adapted insuch a way.

If the temporal distance detection unit 13 is adapted for keeping trackof the temporal distance between successive images and for deriving aprediction of the temporal distance between the current image and theprevious image, it is preferred that the motion estimation unit 11 isadapted for including the one or more substantially doubled candidatemotion vectors in the set of candidate motion vectors in dependence ofthe predicted temporal distance between the current image and theprevious image. For example, in this embodiment, the motion estimationunit 11 is adapted for including the one or more substantially doubledcandidate motion vectors in the set of candidate motion vectors only ifa doubling of the temporal distance between the current image and theprevious image is predicted with a high probability.

In case that a doubling of the temporal distance between the currentimage and the previous image is predicted with a very high probability,it may be advantageous to sacrifice some regular candidate motionvectors for a larger number of substantially doubled motion vectors. Forexample, a set of candidate descriptors that includes an larger numberof substantially doubled candidate motion vectors in the set ofcandidate motion vectors could be: {S, (−1,0), F, D}, {S, (0,1), F, R},{T, (0,0), F, D}, {T, (2,2), F, R}, {S, (−1,0), T, D}, {S, (0,1), T, R},{S, (0,1), F, D}. In general, an optimal set of candidate motion vectorsmay be selected based on thresholds settings adapted to the temporaldistance prediction probability.

In addition, a further optimization can be achieved by avoiding tocalculate the match error for a certain candidate vector more than once(coincidentally, a substantially doubled candidate motion vector may beequal to one of the regular candidate motion vectors).

FIG. 3 shows exemplarily a block diagram of a motion estimation device20 for dealing with missing input images according to a secondembodiment of the invention. The device for motion estimation 20comprises a motion estimation unit 21, a motion vector storage unit 22,and, optionally, a temporal distance detection unit 23. The motionestimation unit 21 is adapted for estimating a current motion vector foran area of a current image by selecting a best motion vector from a setof temporal and/or spatial candidate motion vectors as the currentmotion vector.

The selecting of the current motion vector may comprise calculatingmatch errors for the respective candidate motion vectors and choosingthe current motion vector from the set of temporal and/or spatialcandidate motion vectors by comparing the match errors of the respectivecandidate motion vectors. The temporal and/or spatial candidate motionvectors are typically calculated from a limited number of previouslyestimated motion vectors and, optionally, additional (pseudo random)update vectors. The calculation of the match error may comprise thecalculation of a block matching error, for example, a cross-correlation(CC), a sum of absolute differences (SAD), a mean-squared-error (MSE),or some comparable error measure, if the image area for which thecurrent motion vector is estimated is a block of pixels, but it can alsocomprise the calculation of other error metrics for more general groupsof pixels, for example, groups of pixels representing structuralelements of objects within the current image.

According to the invention, the motion estimation unit 21 is furtheradapted for substantially doubling one or more of the candidate motionvectors and for including the one or more substantially doubledcandidate motion vectors in the set of candidate motion vectors. Themotion estimation unit 21 is further adapted for substantially halvingone or more candidate motion vectors and for including the one or moresubstantially halved candidate motion vectors in the set of candidatemotion vectors.

In this embodiment, the including of the one or more substantiallydoubled or halved candidate motion vectors in the set of candidatemotion vectors may be implemented by a set of candidate descriptors ofthe form {origin, location, random, modifier}, wherein possible valuesof the additional “modifier” are regular (R), double (D) or half (H),indicating a regular candidate motion vector, a substantially doubledcandidate motion vector, or a substantially halved candidate motionvector. An example of a set of candidate descriptors that includes anadditional substantially doubled candidate motion vector and anadditional substantially halved candidate motion vector in the set ofcandidate motion vectors would be: {S, (−1,0), F, R}, {S, (0,1), F, R},{T, (0,0), F, R}, {T, (2,2), F, R}, {S, (−1,0), T, R}, {S, (0,1), T, R},{S, (0,1), F, D}, {S, (0,1), F, H}. However, it would also be possibleto include one or more substantially doubled or halved candidate motionvector instead of regular candidate motion vectors in order to notincrease the computational effort required for the motion estimation.

The motion vector storage unit 22 is adapted for storing the currentmotion vector. In case that one of the one or more substantially doubledor halved candidate motion vectors is selected as the current motionvector, the motion estimation unit 21 stores the substantially doubledor halved candidate motion vector as the current motion vector in themotion vector storage unit. The stored current motion vector istypically used for motion compensation as well as for calculating atemporal candidate motion vector for an image area in the neighborhoodof the current image area in a subsequent image and/or for calculating aspatial candidate motion vector for a different image area in thecurrent image.

The optional temporal distance detection unit 23 is adapted fordetecting a doubling or halving of the temporal distance between thecurrent image and the previous image. In this embodiment, the optionaltemporal distance detection unit 23 is adapted for detecting a doublingor halving of the temporal distance between the current image and theprevious image (relative to the temporal distance between the previousimage pair) by comparing the lengths of current motion vectors forrespective areas of the current image with the lengths of previousmotion vectors for related areas of the previous image. Herein, the term“related area” may relate to the same location in the previous image or,alternatively, may relate to a location in the previous image that isshifted by the current motion vector with respect to a respective areaof the current image. In other embodiments, however, otherheuristics/methods for detecting a doubling or halving of the temporaldistance between the current image and the previous image may be used.

In this embodiment, the temporal distance detection unit 23 is furtheradapted for keeping track of the temporal distance between successiveimages and for deriving a prediction of the temporal distance betweenthe current image and the previous image. In other embodiments, however,the temporal distance detection unit 23 does not have to be adapted insuch a way.

If the temporal distance detection unit 23 is adapted for keeping trackof the temporal distance between successive images and for deriving aprediction of the temporal distance between the current image and theprevious image, it is preferred that the motion estimation unit 21 isadapted for including the one or more substantially doubled or halvedcandidate motion vectors in the set of candidate motion vectors independence of the predicted temporal distance between the current imageand the previous image. For example, in this embodiment, the motionestimation unit 21 is adapted for including the one or moresubstantially doubled candidate motion vectors in the set of candidatemotion vectors only if a doubling of the temporal distance between thecurrent image and the previous image (relative to the temporal distancebetween the previous image pair) is predicted with a high probabilityand for including the one or more substantially halved candidate motionvectors in the set of candidate motion vectors only if a halving of thetemporal distance between the current image and the previous image(relative to the temporal distance between the previous image pair) ispredicted with a high probability.

In case that a doubling or halving of the temporal distance between thecurrent image and the previous image is predicted with a very highprobability, it may be advantageous to sacrifice some regular candidatemotion vectors for a larger number of substantially doubled or halvedmotion vectors. For example, a set of candidate descriptors thatincludes an larger number of substantially doubled candidate motionvectors in the set of candidate motion vectors could be: {S, (−1,0), F,D}, {S, (0,1), F, R}, {T, (0,0), F, D}, {T, (2,2), F, R}, {S, (−1,0), T,D}, {S, (0,1), T, R}, {S, (0,1), F, D}. Likewise, a set of candidatedescriptors that includes an larger number of substantially halvedcandidate motion vectors in the set of candidate motion vectors couldbe: {S, (−1,0), F, H}, {S, (0,1), F, R}, {T, (0,0), F, H}, {T, (2,2), F,R}, {S, (−1,0), T, H}, {S, (0,1), T, R}, {S, (0,1), F, H}. In general,an optimal set of candidate motion vectors may be selected based onthresholds settings adapted to the temporal distance predictionprobability.

In addition, a further optimization can be achieved by avoiding tocalculate the match error for a certain candidate motion vector morethan once (coincidentally, a substantially doubled or halved candidatemotion vector may be equal to one of the regular candidate motionvectors or to one of the other substantially doubled or halved candidatemotion vectors).

FIG. 4 shows exemplarily a block diagram of a motion estimation andcompensation device 100 for dealing with missing input images accordingto a third embodiment of the invention. The device for motion estimationand compensation 100 comprises the device for motion estimationaccording to any of the above-described embodiments of the invention.The device for motion estimation and compensation 100 further comprisesa motion compensation unit 30 adapted for compensating the motionbetween the group of pixels of the current image and a correspondinggroup of pixels of a previous image using the stored current motionvector.

In another embodiment, the motion compensation unit 30 is furtheradapted for doubling the length of the stored current motion vector incase that the temporal distance detection unit 13 detected a doubling ofthe temporal distance between the current image and the previous image.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive; theinvention is not limited to the disclosed embodiments.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure, and theappended claims.

In the claims, the word “comprising” does not exclude other elements orsteps, and the indefinite article “a” or “an” does not exclude aplurality. A single device or other unit may fulfill the functions ofseveral items recited in the claims. The mere fact that certain measuresare recited in mutually different dependent claims does not indicatethat a combination of these measured cannot be used to advantage.

A computer program may be stored/distributed on a suitable medium, suchas an optical storage medium or a solid-state medium supplied togetherwith or as part of other hardware, but may also be distributed in otherforms, such as via the Internet or other wired or wirelesstelecommunication systems.

Any reference signs in the claims should not be construed as limitingthe scope.

What is claimed is:
 1. A motion estimation unit for estimating a currentmotion vector for an area of a current video image, comprising: moduledetermining a set of candidate motion vectors; a temporal distancedetection unit which determines whether the temporal distance between acurrent image and a previous image has doubled due to a missing inputimage; module doubling at least one candidate motion vector in the setof candidate motion vectors if the temporal distance detection unitdetermines that the temporal distance between a current image and aprevious image has doubled; module including the at least one doubledcandidate motion vector in an updated set of candidate motion vectors ifthe temporal distance detection unit determines that the temporaldistance between a current image and a previous image has been doubled;module estimating the current motion vector from the updated set ofcandidate motion vectors by calculating a match error only once for eachcandidate motion vector and comparing the calculated match errors; andmodule halving the candidate motion vectors and including halvedcandidate motion vectors in the updated set of candidate motion vectors.2. The motion estimation unit of claim 1, wherein the device furthercomprises a motion vector storage unit that stores a regular candidatemotion vector as a current motion vector in the motion vector storageunit, in case one of the at least one doubled candidate motion vector isselected as the estimated current motion vector.
 3. The motionestimation unit of claim 2, wherein the temporal distance detection unitdetects a doubling of the temporal distance between the current imageand the previous image by analyzing a number of doubled candidate motionvectors that are selected as estimated current motion vectors forrespective areas of the current image.
 4. The motion estimation unit ofclaim 2, wherein the temporal distance detection unit keeps track of thetemporal distance between successive images and derives a prediction ofthe temporal distance between the current image and the previous image.5. The motion estimation unit of claim 1, further comprising a motionvector storage unit for storing a regular candidate motion vector as acurrent motion vector in the motion vector storage unit in case one ofthe at least one halved candidate motion vectors is selected as theestimated current motion vector.
 6. The motion estimation unit of claim5, wherein the motion estimation unit further comprises a temporaldistance detection unit for detecting a halving of the temporal distancebetween a current image and a previous image.
 7. The motion estimationunit of claim 6, wherein the temporal distance detection unit detects ahalving of the temporal distance between the current image and theprevious image by comparing lengths of the current motion vectors forrespective areas of the current image with lengths of previous motionvectors for related areas of the previous image.
 8. The motionestimation unit of claim 6, wherein the temporal distance detection unitkeeps track of the temporal distance between the current image and theprevious image and derives a prediction of the temporal distance betweenthe current image and the previous image.
 9. The motion estimation unitof claim 8, wherein the motion estimation unit is further adapted forincluding the one or more substantially doubled or halved candidatemotion vectors in the set of candidate motion vectors in dependence ofthe predicted temporal distance between the current image and theprevious image.
 10. The motion estimation unit of claim 2, furthercomprising a motion compensation unit adapted for compensating themotion between the area of the current image and a corresponding area ofa previous image using the stored regular candidate motion vector if thestored regular candidate motion vector is doubled and the temporaldistance detection unit has detected a doubling of the temporal distancebetween the adjacent video frames.
 11. The motion estimation unit ofclaim 5, further comprising a motion compensation unit for compensatingmotion between an area of the current image and a corresponding area ofa previous image using the stored regular candidate motion vector.
 12. Amethod for estimating a current motion vector for an area of a currentvideo image, comprising: determining a set of candidate motion vectors;determining a temporal distance between adjacent video frames; doublingat least one candidate motion vector in the set of candidate motionvectors if the distance between adjacent video frames is determined tobe double due to a missing input image; including the at least onedoubled candidate motion vector in an updated set of candidate motionvectors if the distance between adjacent video frames is doubled;estimating the current motion vector from the updated set of candidatemotion vectors by calculating a match error only once for each candidatemotion vector and comparing the calculated match errors; halving thecandidate motion vectors; and including the halved candidate motionvectors in the updated set of candidate motion vectors.